Earlier, CONTAINER_METADATA_SIZE was defined as 1MB to estimate a ContainerInfo object. A ContainerID proto is ~8–12 bytes. With an IPC max of 128MB, the old code limited batches to floor(128MB / 1MB) = 128 containers per call. The new code does the same — 128 IDs per call — when it could safely fetch floor(128MB / 12 bytes) ≈ 5.5 million IDs per call. This means the change may actually increase the number of RPCs instead of reducing them. So I think, we should test with large set of container Ids specially when a cluster will have 4-5 millions CLOSED containers and record the SCM latency and impact. Because that was the whole objective behind why this JIRA was raised. And based on impact, we should think of innovative way to handle impact on SCM as well as rpc message length also should not exceed default 128 MB.

I propose limiting the number of container IDs we fetch back in a single message to 500K. This would equate to 16MB which is well within the RPC message size limit and should be safe enough for SCM to handle.

I also want to introduce a config where we can reduce the number of container ids fetched if we need to fine tune due to memory constraints

The "~16MB heap" is computed as 500K × ~32 bytes per JVM ContainerID object. But the wire size is 500K × 12 bytes = 6MB. Check my comment in other place.

The "~16MB heap" is computed as 500K × ~32 bytes per JVM ContainerID object. But the wire size is 500K × 12 bytes = 6MB. These are different things and the comment doesn't distinguish them. Operators tuning this config for RPC limits care about wire size; operators tuning for memory care about heap size. Clarify both: e.g., "~6MB on the wire, ~16MB JVM heap per batch". So based on wire size, number of containers in single rpc call also can be more in default ?

The "~16MB heap" is computed as 500K × ~32 bytes per JVM ContainerID object. But the wire size is 500K × 12 bytes = 6MB. Check my comment in other place.